Display control apparatus, method, recording medium, and vehicle

ABSTRACT

A method capable of causing a driver to recognize multiple objects to which attention should be paid without requiring sight line shift of the driver is for a video system including a recognition unit that recognizes a moving body existing in a foreground of a user and a display source that displays a certain image generated on the basis of a result of the recognition by the recognition unit on a display medium. The method includes determining whether two or more moving bodies exist in the foreground of the user on the basis of an input data from the recognition unit; and controlling the display source, if two or more moving bodies exist, so as to generate the certain image representing a virtual line directed to each of the moving bodies from a certain position or a certain axis and display the certain image on the display medium.

BACKGROUND

1. Technical Field

The present disclosure relates to a technology to control display ofinformation for alerting the driver of a vehicle.

2. Description of the Related Art

Advanced Driver Assistance Systems (ADAS) for alerting the drivers ofvehicles are currently in widespread use. Applications for the ADASinclude Lane Departure Warning (LDW), Forward Collision Warning (FCW),and Pedestrian Collision Warning (PCW). In the ADAS, travelingenvironment including white lines, preceding vehicles and vehiclesapproaching from the behind, pedestrians on the traveling directions,and so on are detected with sensors, such as on-vehicle cameras oron-board radars, to display information based on the results of thedetection. The information is displayed on, for example, meter displays,head-up displays (HUDs), head-mounted displays or helmet-mounteddisplays (HMDs), or Smart Glasses.

As is known as a phenomenon “inattentional blindness” in general, it isdifficult for the drivers to recognize objects other than the objects towhich the drivers give attention if the objects simultaneously exist.Therefore, the ADAS in which information about the objects, such asvehicles on the front side and/or pedestrians, detected with the sensorsis overlapped and displayed on the foreground of the drivers has beendeveloped. With the ADAS, it is possible to present the informationabout the objects which the drivers do not aggressively search for orwhich the drivers do not pay attention to the drivers, thus realizingeffective driver assistance.

Information presentation apparatuses in related art, for example,generate two-dimensional feature maps to generate saliency mapsindicating the focuses of attention from the two-dimensional featuremaps. The two-dimensional feature maps result from mapping of simplevisual features in an initial phase of visual information processing ofhuman beings on the same spatial coordinate system as that of cameraimages on the basis of features, such as areas or centroids, of objectsin the camera images. The focuses of attention are areas to which thedrivers of vehicles are easy to pay attention and, for example, areareas where images of the preceding vehicles appear. If the objectswhich the drivers do not pay attention appear, the informationpresentation apparatuses in the related art re-create the saliency mapsto move the focuses of attention to the areas where the objects appear.The information presentation apparatuses in the related art present theinformation to the drivers on the basis of the focuses of attention thatare moved (for example, refer to Japanese Unexamined Patent ApplicationPublication No. 2011-8772).

However, the technology in Japanese Unexamined Patent ApplicationPublication No. 2011-8772 has a problem in that the movement of thefocuses of attention triggers sight line shifts of the drivers toincrease the burdens on the drivers. The technology in JapaneseUnexamined Patent Application Publication No. 2011-8772 also has aproblem in that the movement of the focuses of attention imposes afurther burden on the drivers because the drivers pay attention to thefocuses of attention that are moved to cause oversight.

SUMMARY

Thus, a non-limiting exemplary embodiment of the present disclosureprovides a method for display control capable of causing a driver torecognize multiple objects to which attention should be paid withoutrequiring the sight line shift of the driver.

According to an embodiment of the present disclosure, a method for avideo system including a recognition unit that recognizes a moving bodyexisting in a foreground of a user and a display source that displays acertain image generated on the basis of a result of the recognition bythe recognition unit on a display medium includes determining whethertwo or more moving bodies exist in the foreground of the user on thebasis of an input data from the recognition unit; and controlling thedisplay source, if it is determined that two or more moving bodiesexist, so as to generate the certain image representing a virtual linedirected to each of the two or more moving bodies from a certainposition or a certain axis and display the generated certain image onthe display medium.

It should be noted that general or specific embodiments may be realizedas a system, a method, an integrated circuit, a computer program,storage media, or any elective combination thereof.

According to the present disclosure, it is possible to cause a driver torecognize multiple objects to which attention should be paid withoutrequiring the sight line shift of the driver.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary configuration of aprojection apparatus according to first to fourth embodiments of thepresent disclosure;

FIG. 2 is a flowchart illustrating an exemplary operational process ofan information presentation apparatus according to the first embodimentof the present disclosure;

FIG. 3 illustrates an example of a foreground in informationpresentation in the information presentation apparatus according to thefirst embodiment of the present disclosure;

FIGS. 4A to 4D illustrate four examples of the information presentationin the information presentation apparatus according to the firstembodiment of the present disclosure;

FIGS. 5A to 5D illustrate other four examples of the informationpresentation in the information presentation apparatus according to thefirst embodiment of the present disclosure;

FIGS. 6A to 6D illustrate other four examples of the informationpresentation in the information presentation apparatus according to thefirst embodiment of the present disclosure;

FIGS. 7A to 7D illustrate other four examples of the informationpresentation in the information presentation apparatus according to thefirst embodiment of the present disclosure;

FIGS. 8A to 8D illustrate other four examples of the informationpresentation in the information presentation apparatus according to thefirst embodiment of the present disclosure;

FIG. 9 is a flowchart illustrating an exemplary operational process ofan information presentation apparatus according to the second embodimentof the present disclosure;

FIG. 10 illustrates an example of a foreground in the informationpresentation in the information presentation apparatus according to thesecond embodiment of the present disclosure;

FIGS. 11A to 11D illustrate four examples of the informationpresentation in the information presentation apparatus according to thesecond embodiment of the present disclosure;

FIG. 12 is a flowchart illustrating an exemplary operational process ofan information presentation apparatus according to the third embodimentof the present disclosure;

FIG. 13 is a timing chart indicating an example of informationacquisition timing in the information presentation apparatus accordingto the third embodiment of the present disclosure;

FIGS. 14A to 14C illustrate three examples of the informationpresentation in the information presentation apparatus according to thethird embodiment of the present disclosure;

FIG. 15 is a flowchart illustrating an exemplary operational process ofan information presentation apparatus according to the fourth embodimentof the present disclosure;

FIG. 16 illustrates an example of a foreground in the informationpresentation in the information presentation apparatus according to thefourth embodiment of the present disclosure; and

FIG. 17 is a block diagram illustrating an exemplary hardwareconfiguration of the information presentation apparatuses according tothe first to fourth embodiments of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will herein be described withreference to the attached drawings.

First Embodiment

A first embodiment of the present disclosure will now be described.

FIG. 1 is a block diagram illustrating an exemplary configuration of avideo system 10 according to the first embodiment. Referring to FIG. 1,the video system 10 includes a display control apparatus 100, a displaysource 400, and a recognition unit 200.

The recognition unit 200 acquires a foreground image representing theforeground of a user (hereinafter simply referred to as “foreground”) torecognize a moving body included in the foreground image. Since theforeground image representing the foreground is acquired by a cameraprovided on the front side of a vehicle, the foreground of the user isthe foreground in the direction in which the vehicle is traveling.Specifically, the recognition unit 200 acquires the foreground imagethat is captured by the camera (not illustrated) and that represents theforeground. When the user is the driver of the vehicle, the camera ismounted on the front side of the vehicle to capture an image of theforeground of the driver. When the user is a pedestrian, the camera ismounted in a wearable terminal to capture an image of the foreground ofthe pedestrian.

After acquiring the foreground image captured by the camera, therecognition unit 200 determines whether a predetermined object isincluded in the foreground image to recognize the moving body. Therecognition unit 200 extracts features included in the image to performobject recognition on the basis of the extracted features. Sincetechnologies concerning the object recognition are known, a detaileddescription of the technologies is omitted herein.

The display control apparatus 100 controls image data projected by thedisplay source 400 on the basis of an input data from the recognitionunit 200. Although the image data is described so as to be generated inthe display source 400 in the following description, the image data maybe generated by the display control apparatus 100 or another componentthat is not illustrated.

The display source 400 projects the image data onto a display (notillustrated). The display source 400 has a projector function anddirectly projects the image data onto the display (also referred to as adisplay medium). The display source 400 may use, for example, a displayfunction on the display or an output function with a hologram, insteadof the projector function. When the output function with the hologram isused, a method may be adopted in which part of parallel light beams ledby internal total reflection of a light guide is emitted to cause apassenger to visually recognize a virtual image. The light guideperforms the internal total reflection of the parallel light beamsmeeting an internal total reflection condition of the light guide tolead the light. Although the image data is not directly projected in themethod using the light guide, unlike the projector, the term“projection” may be used for the description also in the method usingthe light guide, as in the projector method, for convenience.

When a light transmission display (an optical see-through display) orthe like is used, the display source (hereinafter also referred to as adisplay unit) 400 displays the image data on the display. The image datamay be described so as to be displayed on the display forsimplification, also in a case in which a virtual-image projectiondisplay or the light transmission display is used.

For example, the HUD, the HMD, or the Smart Glasses are used as thedisplay. The HUD may be, for example, a window shield of the vehicle ora glass plate or a plastic plate separately provided. The window shieldmay be a windshield or a glass of side window or a rear glass of thevehicle.

The display may be any of the virtual-image projection display, a retinaprojection display, the light transmission display (the opticalsee-through display), a video transmission display (a video see-throughdisplay), and a non-transmission display. An image to be displayed (adisplay image) may be the virtual image or a real image. Research anddevelopment of an on-vehicle HUD is advanced as the light transmissiondisplay (Hye Sun Park, Min Woo Park, Kwang Hee Won, Kyong-Ho Kim, andSoon Ki Jung, “In-Vehicle AR-HUD System to Provide Driving-SafetyInformation”, ETRI Journal, Volume 35, Number 6, December 2013). Forexample, the light transmission display may be stuck on the internalsurface of the window shield of the vehicle or may be incorporated intothe window shield in a process to cast the window shield of the vehicle.

Although the display is described as a virtual-image projection HUD inthe following description, the display is not limited to thevirtual-image projection HUD and may be any of the various displaysdescribed above. When the light transmission display is used, instead ofthe HUD, the display image displayed on the display is recognized not asthe virtual image but as the real image by the user.

The video system 10 may include the display described above.

The image data generated in the video system 10 includes displayinformation to be presented to the user of the apparatus. The displayinformation is displayed on the display as part of the image data. Thedisplay information that is displayed is visually recognized by the useras the virtual image. Since the principle that the display informationprojected onto the display is visually recognized by the user as thevirtual image is well known, a description of the principle is omittedherein.

The display information may be projected onto the display so as to beoverlapped on the foreground, instead of being projected onto thedisplay as part of the image data. In this case, the display controlapparatus 100 may control only the display information and the displaysource 400 may project only the display information onto the display. Anexample will be described below in which a virtual line, which is anexample of the display information, is projected onto the display so asto be overlapped on the foreground.

Although an example will be described below in which the user is apassenger of the vehicle, particularly, the user is the driver of thevehicle, the user is not limited to a passenger of the vehicle or thedriver of the vehicle. For example, when the HMD is adopted as thedisplay, the user may be a passenger of the vehicle or a pedestrian. Inother words, the video system 10 may an on-vehicle apparatus, may be anapparatus brought into the vehicle for usage, or may be an apparatusworn by the user during walking.

An exemplary configuration of the display control apparatus 100 of thefirst embodiment will now be described. Referring to FIG. 1, the displaycontrol apparatus 100 includes a determination unit 101 and a controlunit 102. The determination unit 101 determines whether two or moremoving bodies exist in the foreground on the basis of an input data fromthe recognition unit 200. If the determination unit 101 determines thattwo or more moving bodies exist, the control unit 102 controls thedisplay source 400 so as to generate a certain image representing thevirtual line directed to each of the two or more moving bodies from acertain position in the projection onto the display medium and projectthe generated certain image onto the display medium. This will bespecifically described. The display control apparatus 100 may include aconnecting unit with the recognition unit 200 and a connecting unit withthe display source 400.

The determination unit 101 acquires information concerning each movingbody from the recognition unit 200. The information concerning eachmoving body at least includes the presence of the moving body. Theinformation concerning each moving body may include position informationin the foreground. The determination unit 101 recognizes the number ofmoving bodies existing in the foreground on the basis of the informationconcerning each moving body. The determination unit 101 determineswhether one moving body or two or more moving bodies exist in theforeground and outputs the result of the determination to the controlunit 102.

If the determination unit 101 determines that two or more moving bodiesexist in the foreground, the control unit 102 controls the displaysource 400 so as to generate the certain image and project the generatedcertain image onto the display. The certain image generated in thedisplay source 400 is visually recognized by the user as the virtualimage when the certain image is projected onto the display. The certainimage is the virtual line directed to each of the two or more movingbodies from the certain position on the display. In other words, thevirtual line is an image visually recognized so as to be directed toeach of the two or more moving bodies from the certain position on thedisplay when the user watches the foreground.

An exemplary operation of the display control apparatus 100 of the firstembodiment will now be described. FIG. 2 is a flowchart illustrating anexemplary operational process of the display control apparatus 100 ofthe first embodiment.

An example of the foreground illustrated in FIG. 3 is used in thedescription of the exemplary operation.

FIG. 3 illustrates an example of the foreground in a state in which thecertain image representing the virtual line is not projected onto thedisplay. For example, a road, buildings, and pedestrians on both sidesof the road exist in a foreground 300 in FIG. 3 as a scene on the frontside of the vehicle. A position 301 indicates the position of apedestrian (an example of the moving body) who is walking or running onthe left side in the traveling direction of the vehicle and a position302 indicates the position of a pedestrian (an example of the movingbody) who is walking or running on the right side in the travelingdirection of the vehicle (refer to circles in FIG. 3). Referencenumerals 311 to 313 in FIG. 3 will be described below.

Referring to FIG. 2, in Step S201, the recognition unit 200 acquires theforeground image representing the foreground to recognize any movingbody included in the foreground. It is assumed here that one or moremoving bodies exist in the foreground.

Steps S101 to S103 are performed by the display control apparatus 100.In Step S101, the determination unit 101 in the display controlapparatus 100 determines whether two or more moving bodies exist in theforeground on the basis of the foreground image. If the determinationunit 101 determines that two or more moving bodies exist in theforeground (YES in Step S101), the process goes to Step S102.

In Step S102, the control unit 102 in the display control apparatus 100controls the display source 400 so as to generate the certain image,which is the virtual line directed to each of the multiple moving bodiesfrom a certain position or a certain axis in the projection onto thedisplay medium (the display), and project the generated certain imageonto the display.

If the determination unit 101 determines that two or more moving bodiesdo not exist in the foreground, that is, the determination unit 101determines that one moving body exists in the foreground (NO in StepS101), the process goes to Step S103.

In Step S103, the control unit 102 controls the display source 400 so asto generate the certain image, which is the virtual line directed to themoving body from a certain position or a certain axis in the projectiononto the display, and project the generated certain image onto thedisplay. The control unit 102 may control the display source 400 so asto generate the certain image representing the position of the movingbody in the projection onto the display and project the generatedcertain image onto the display. The certain image representing theposition of the moving body is displayed in, for example, a circularshape below the moving body. The shape of the certain image is notlimited to the circle and the certain image is not limitedly displayedbelow the moving body.

In Step S301, the display source 400 generates the certain image on thebasis of an instruction from the control unit 102. The display source400 projects the generated certain image onto the display. The virtualimage corresponding to the position of each moving body existing in theforeground is displayed in the above manner.

FIG. 4A to FIG. 4D illustrate examples of how the certain imagesgenerated in the display source 400 are projected onto the display. Thecertain images generated in the display source 400 will now be describedwith reference to FIG. 4A to FIG. 4D.

If the determination unit 101 determines that one moving body exists inthe foreground, the projection of the certain image onto the displayproduces the foreground illustrated in FIG. 4A. In the example in FIG.4A, an image 401 is a virtual line directed from a point 405 to themoving body. The virtual line is overlapped on the foreground and isvisually recognized by the user as the virtual image.

If the determination unit 101 determines that two moving bodies exist inthe foreground, the projection of the certain images onto the displayproduces the foreground illustrated in FIG. 4B. In the example in FIG.4B, each of the image 401 and an image 402 is a virtual line directedfrom the point 405 to each moving body. The virtual lines are overlappedon the foreground and are visually recognized by the user as the virtualimages.

When the number of moving bodies is increased by one from the stateillustrated in FIG. 4B, the determination unit 101 determines that threemoving bodies exist and the foreground illustrated in FIG. 4C isproduced. In the example in FIG. 4C, an image 403 is a virtual linedirected from the point 405 to the moving body that is added. Thevirtual lines are overlapped on the foreground and are visuallyrecognized by the user as the virtual images.

When the number of moving bodies is decreased by one from the state inFIG. 4C and the positions of the remaining moving bodies are overlappedwith each other, the foreground illustrated in FIG. 4D is produced. Inthe example in FIG. 4D, the image 401 and the image 402 in FIG. 4C arenot overlapped on the foreground and only the image 403 is overlapped onthe foreground as the virtual line.

In the examples in FIG. 4A to FIG. 4D, the images 401 to 403 and thepoint 405 are the certain images generated in the display source 400,that is, the virtual lines. The user is capable of recognizing thenumber of branches of the virtual lines directed from the point 405 tothe moving bodies and the directions of the virtual lines only by payingattention to the periphery of the point 405, which is the certainposition, to intuitively understand the number and the positions of themoving bodies existing in the foreground. The point 405 may not beprojected onto the display.

The virtual lines illustrated in FIG. 4A to FIG. 4D are only examples.The virtual lines may have wedge shapes, as illustrated in FIG. 5A toFIG. 5D, and may have thicknesses that are varied depending on thepositions or the states of the moving bodies. In this case, it ispossible for the user to intuitively understand the distance to eachmoving body and the risk of collision with the moving body by increasingthe thickness of the virtual line at the certain position, for example,with the decreasing distance from the own vehicle to the moving body orwith the increasing relative traveling speed of the moving body relativeto the own vehicle. Since the thickness of the virtual line is decreasedwith the increasing distance from the certain position to the movingbody, wasteful display is reduced for the user who pays attention to theperiphery of the point 405. As a result, the user is not needed to payattention to the virtual lines more than needs.

The starting point of the virtual line directed to each moving body maynot be a certain position and may be a certain axis, as illustrated inFIG. 6A to FIG. 6D. In this case, for example, the user is capable ofintuitively understanding the number and the positions of the movingbodies existing in the foreground by paying attention only to theperiphery of the certain axis in the foreground. Alternatively, thecertain axis may be projected onto the display as an image, asillustrated in FIG. 7A to FIG. 7D. In this case, it is possible for theuser to define the range to which the user should pay attention. Theuser is capable of intuitively understanding the number and thepositions of the moving bodies existing in the foreground by payingattention only to the periphery of the certain axis that is projected.

The certain position, which is the starting point of the virtual linedirected to each moving body, may be a point of view of the user on thedisplay medium, for example, which is calculated from the orientationand the line of sight of the user detected by an on-vehicle cameramounted in the vehicle. In this case, the user is capable of intuitivelyunderstanding the number and the positions of the moving bodies existingin the foreground regardless of where the point of view of the userexists.

The certain position, which is the starting point of the virtual linedirected to each moving body, may be an end point of the precedingvehicle or a point on the contour of the preceding vehicle. For example,the left rear wheel of the preceding vehicle may be set as the certainposition, which is the starting point of the virtual line directed tothe moving body existing on the left side in the traveling direction ofthe own vehicle, and the right rear wheel of the preceding vehicle maybe set as the certain position, which is the starting point of thevirtual line directed to the moving body existing on the right side inthe traveling direction of the own vehicle. In this case, the user iscapable of intuitively understanding the number and the positions of themoving bodies existing on the left and right sides in the foregroundwhile paying attention to the preceding vehicle.

The certain position, which is the starting point of the virtual lines,may be varied depending on the type or the state of each moving body.For example, the certain position, which is the starting point of thevirtual lines, may be varied depending on whether the type of the objectis a pedestrian or a bicycle. In this case, since the moving bodies aregrouped as the pedestrians and the bicycles for display, the user iscapable of understanding the number and the positions of the movingbodies of the type to which close attention should be paid by priority.Alternatively, for example, the certain position, which is the startingpoint of the virtual lines, may be varied depending on whether themoving body is approaching the own vehicle or is moving apart from theown vehicle. Alternatively, for example, the moving bodies may begrouped into multiple groups on the basis of the moving speeds and thecertain position, which is the starting point of the virtual lines, maybe varied for each group. In the above cases, the user is capable ofunderstanding the number and the positions of the moving bodies that areapproaching the own vehicle and that have higher risk of collision orthe moving bodies having higher moving speeds by priority.

As illustrated in FIG. 5A to FIG. 5D, the color of the virtual lines maybe lightened or the transparency of the virtual lines may be increasedwith the increasing distance from the point 405, which is the certainposition, to each moving body. Alternatively, a combination of the abovemethods may be adopted. This reduces wasteful display for the user whopays attention to the periphery of the point 405. As a result, the useris not needed to pay attention to the virtual lines more than needs.

The virtual line may be a perpendicular extending from the position ofeach moving body to a vertical line close to the point of view of theuser in the foreground, for example, the vertical line through thecenter of the foreground, as illustrated in FIG. 6A to FIG. 6D. If theuser focuses on the foreground when the user simultaneously watches theforeground and the virtual lines, the virtual lines may be horizontallydoubled. However, since the virtual lines extend horizontally, it isdifficult for the user to recognize the virtual lines that are doubled.A vertical line connecting the perpendiculars extending to the verticalline through the center of the foreground may be visualized. The user iscapable of intuitively understand the number and the positions of themoving bodies existing in the foreground by paying attention only to thevertical line connecting the perpendiculars extending to the verticalline through the center of the foreground.

As illustrated in FIG. 3, it is assumed that a basic sight line position(vanishing point) 311, a lower sight line position 312, and an uppersight line position 313 are defined in the front direction of thevehicle (refer to triangles in FIG. 3). The basic sight line position(vanishing point) 311 is a position to which the user normally paysattention in the foreground. The lower sight line position 312 is aposition close to the vehicle on a line connecting the basic sight lineposition 311 to the vehicle. The upper sight line position 313 is aposition far from the vehicle on the line connecting the basic sightline position 311 to the vehicle. The lower sight line position 312 andthe upper sight line position 313 are not desirably overlapped with thepreceding vehicle on the front side of the vehicle in the foreground andare desirably closer to the basic sight line position (vanishing point)311 as much as possible. The basic sight line position 311, the lowersight line position 312, and the upper sight line position 313 are notprojected onto the display.

FIG. 8A illustrates an example in which the moving bodies are directlyconnected to each other with the virtual line without using eachposition illustrated in FIG. 3. For example, directly connecting the twomoving bodies closest to the own vehicle on the right side and the leftside on the front side of the vehicle with the virtual lines allows theuser to easily recognize the presence of the moving body to which theuser should pay most attention. The virtual lines are reconnected eachtime the moving body closest to the own vehicle is switched.

FIG. 8B to FIG. 8D illustrate examples in which the moving bodies areconnected to each other with the virtual lines by using the respectivepositions illustrated in FIG. 3. Specifically, FIG. 8B illustrates anexample in which the position 301 of a first moving body is connected tothe position 302 of a second moving body with the virtual lines via thebasic sight line position 311. FIG. 8C illustrates an example in whichthe position 301 of the first moving body is connected to the position302 of the second moving body with the virtual lines via the lower sightline position 312. FIG. 8D illustrates an example in which the position301 of the first moving body is connected to the position 302 of thesecond moving body with the curved virtual lines via the upper sightline position 313. For example, if the preceding vehicle exists on thefront side of the vehicle when the virtual lines are connected via thebasic sight line position 311, as in FIG. 3B, the virtual lines may beoverlapped and displayed on the preceding vehicle and the user may bedifficult to visually recognize the preceding vehicle and the virtuallines. Connecting the moving bodies with the virtual lines via the lowersight line position 312 or the upper sight line position 313 avoids thedifficulty in the visual recognition of the preceding vehicle and thevirtual lines by the user.

The virtual lines illustrated in FIG. 8A to FIG. 8D are determined bythe control unit 102, as in the virtual lines illustrated in FIG. 4A toFIG. 7D. Although the virtual lines have the linear shapes, the wedgeshapes, or the curved shapes in the first embodiment, as illustrated inFIG. 4A to FIG. 8D, the shapes of the virtual lines are not limited tothe above ones.

In the first embodiment, when it is not possible to present all thevirtual lines under restrictions of, for example, the size of thedisplay, part of the virtual lines may be presented in a range in whichthe virtual lines are capable of being presented.

Although the virtual lines are overlapped and displayed on theforeground in the first embodiment, the virtual lines are not limited tothis. For example, the virtual lines may be overlapped and displayed onthe image data of the foreground.

In the first embodiment, the virtual lines illustrated in FIG. 8A toFIG. 8D may be selectively used depending on the status of the frontside of the vehicle. For example, the virtual line 401 illustrated inFIG. 8A may be displayed when no preceding vehicle exists, the virtualline 403 illustrated in FIG. 8C may be displayed when the vehicular gapbetween the own vehicle and the preceding vehicle is sufficiently kept,and a virtual line 404 illustrated in FIG. 8D may be displayed when thevehicular gap between the own vehicle and the preceding vehicle is notsufficiently kept.

As described above, the display control apparatus 100 of the firstembodiment is characterized in that, when multiple moving bodies existin the foreground, the virtual lines connecting the positions of themultiple moving bodies are overlapped on the foreground for thepresentation to the user. Accordingly, the display control apparatus 100is capable of causing the user to recognize the moving bodies, or themultiple objects to which attention should by paid, without requiringthe sight line shift.

In the above embodiment, when it is determined that two or more movingbodies exist in the foreground, the certain image representing thevirtual line directed to each of the two or more moving bodies from thecertain position or the certain axis in the projection onto the displayis generated and the certain image representing the virtual line isdisplayed on the display. Accordingly, it is possible for the user topay attention to one point, the certain position or the certain axis, torecognize the respective directions in which the two or more movingbodies exist from the one point, for example, compared with a case inwhich images in which the two or more moving bodies are surrounded byrectangular lines are displayed. In other words, when the images inwhich the two or more moving bodies are surrounded by the rectangularlines are displayed, it is necessary for the user to recognize each ofthe two or more moving bodies while directing the line of sight to eachof the images in which the two or more moving bodies are surrounded bythe rectangular lines. For example, when pedestrians exist on theleft-side sidewalk and the right-side sidewalk while the user is drivingthe vehicle, each of the images in which the pedestrians on theleft-side sidewalk and the right-side sidewalk are surrounded by therectangular lines draws attention of the user because the images inwhich the two or more moving bodies are surrounded by the rectangularlines are enhanced. Accordingly, the time during which the line of sightof the user stops is increased at each of the images in which thepedestrians on the left-side sidewalk and the right-side sidewalk aresurrounded by the rectangular lines and the attention to the front sideis reduced to possibly increase the risk.

In contrast, in the first embodiment, since the user is capable ofpaying attention to one point, the certain position or the certain axis,to recognize the respective directions in which the two or more movingbodies exist from the one point, it is possible for the user to acquirethe information about the respective directions in which the two or moremoving bodies exist at once by paying attention to the one point, thecertain position or the certain axis. Accordingly, since the user iscapable of acquiring the information about the respective directions inwhich the two or more moving bodies exist at once in the firstembodiment, it is possible to instantaneously acquire the accurateinformation regardless of the number of the moving bodies. Consequently,it is possible for the user to acquire the information concerning thesurrounding moving bodies while paying sufficient attention to the frontside, for example, during the driving to safely drive the vehicle.

Although the position information about the moving bodies acquired bythe determination unit 101 is recognized on the basis of the foregroundimage acquired by the recognition unit 200 in the above description, theposition information about the moving bodies is not limitedly recognizedin the above manner. For example, the position information about themoving bodies may be output from a position information generatingapparatus (not illustrated) that generates the position information. Theposition information may be, for example, the position informationindicated by the saliency map extracted from an output signal from avisual sensor, the position information indicating the positions of thebicycle, the pedestrian, and so on subjected to the image recognition,the position information indicating the relative position calculatedfrom the positions of other surrounding moving bodies and the positionof the own vehicle received through inter-vehicle communication, or theposition information indicating the relative position calculated fromthe positions of other surrounding moving bodies and the position of theown vehicle received through road-to-vehicle communication from aroad-side sensor.

Second Embodiment

A second embodiment of the present disclosure will now be described.

The block diagram illustrating an exemplary configuration of a videosystem according to the second embodiment is the same as that of thefirst embodiment illustrated in FIG. 1. Referring to FIG. 1, the videosystem 10 includes the display control apparatus 100, the display source400, and the recognition unit 200. The display control apparatus 100includes the determination unit 101 and the control unit 102. Theoperations of the recognition unit 200 and the display source 400 aresame as those in the first embodiment.

An exemplary operation of the display control apparatus 100 of thesecond embodiment will now be described. FIG. 9 is a flowchartillustrating an exemplary operational process of the display controlapparatus 100 of the second embodiment.

An example of the foreground illustrated in FIG. 10 is used in thedescription of the exemplary operation. The “object” in the secondembodiment has the same meaning as that of the “moving body” in thefirst embodiment. The user of the display control apparatus 100 is thedriver of a vehicle in the second embodiment.

FIG. 10 illustrates an example of the foreground of the driver before apresentation area is presented. The same reference numerals are used inFIG. 10 to identify the same components in FIG. 3. A description of suchcomponents is omitted herein. Referring to FIG. 10, a frame 601indicates the position and the size of a bicycle (an example of a firstobject) that is traveling on the left side in the traveling direction ofthe vehicle and a frame 602 indicates the position and the size of abicycle (an example of a second object) that is traveling on the rightside in the traveling direction of the vehicle (refer to rectangles inFIG. 10).

Referring to FIG. 9, in Step S501, the determination unit 101 acquiresthe position information and size information about the object from therecognition unit 200 as the information concerning the object. The sizeinformation indicates the apparent size of the object in the foreground.The frame 601 is acquired as the position information and the sizeinformation about the first object in the foreground 300 as an examplehere.

In Step S502, the determination unit 101 determines whether thepresentation area that is being presented exists. The presentation areais a collective term of a single presentation area and connectedpresentation areas described below. If the determination unit 101determines that no presentation area that is being presented exists (NOin Step S502), the process goes to Step S503.

In Step S503, the determination unit 101 determines the singlepresentation area on the basis of the position information and the sizeinformation that are acquired. The single presentation area is an areafor presentation to indicate the position of one object. In other words,the single presentation area is a virtual graphic of a certain shape,which is visually recognized as the virtual image by the user so thatthe certain image exists at the position corresponding to the movingbody when the certain image is projected on the display medium. Thesingle presentation area corresponding to the position and the size ofthe first object is determined as an example here. The position wherethe single presentation area is displayed is determined, for example, tobe below the first object, which corresponds to the position of thefirst object, and the single presentation area is determined to have anellipse (refer to a presentation area 701 a in FIG. 11A). The size ofthe single presentation area may be determined so as to reflect the sizeof the first object and the single presentation area may have a shapeother than the ellipse. The single presentation area is not limitedlydisplayed below the object.

In Step S505, the control unit 102 adds representative information tothe presentation area determined by the determination unit 101. Therepresentative information indicates the type of the object and thestate of the object. The control unit 102 supplies the presentation areato the display source 400. Then, the display source 400 projects thepresentation area onto a certain display. This causes the presentationarea, which is the virtual image and which corresponds to the objectexisting in the foreground of the driver, to be displayed. Apredetermined color, blue, is added to the single presentation areacorresponding to the first object as the representative informationabout the bicycle as an example here. As a result, the singlepresentation area corresponding to the first object (for example, thepresentation area 701 a in FIG. 11A) is displayed in blue in theforeground 300.

Then, the process goes back to Step S501. The description of thespecific example will be continued.

In Step S501, the determination unit 101 acquires the positioninformation and the size information about the object. The frame 602 isacquired as the position information and the size information about thesecond object in the foreground 300 as an example here.

In Step S502, the determination unit 101 determines whether thepresentation area that is being presented exists. Since the singlepresentation area corresponding to the first object exists as thepresentation area that is being presented in the foreground 300 as anexample here (YES in Step S502), the process goes to Step S504.

In Step S504, the determination unit 101 determines the connectedpresentation areas on the basis of the position information and the sizeinformation that are acquired. The connected presentation areas are thepresentation areas to connect the positions of multiple objects. Whenthe presentation area has already been presented, the connectedpresentation areas correspond to the single presentation area of anobject the position information and the size information about which arenewly acquired and the virtual lines connecting the positions of themultiple objects. Here, the single presentation area of the secondobject (for example, a presentation area 702 a in FIG. 11A) isdetermined and virtual lines (for example, presentation areas 703 a and704 a in FIG. 11A) connecting the position of the first object to theposition of the second object are determined.

In Step S505, the control unit 102 adds the representative informationto the presentation areas determined by the determination unit 101.Then, the presentation areas are supplied to the display source 400where the presentation areas are projected on the certain display as thevirtual images.

An example of how the single presentation areas and the virtual linesare displayed is illustrated in FIG. 11A. As illustrated in FIG. 11A,the ellipsoidal presentation area 701 a is displayed below the firstobject and the ellipsoidal presentation area 702 a is displayed belowthe second object. The virtual line 703 a is displayed so as to projectfrom part of the periphery of the presentation area 701 a, and thepresentation area 704 a is displayed so as to project from part of theperiphery of the presentation area 702 a. The virtual line 703 a and thevirtual line 704 a connect the position of the first object to theposition of the second object. The predetermined color, blue, is addedto the single presentation areas and the virtual lines as therepresentative information about the bicycles as an example here. Forexample, in the example in FIG. 11A, the presentation areas 701 a and702 a and the virtual lines 703 a and 704 a are displayed in blue.

The presentation of the virtual lines is not limited to the mannerillustrated in the example in FIG. 11A. Other examples of how thevirtual lines are presented will now be described with reference to FIG.10 and FIG. 11B to FIG. 11D. Since the basic sight line position(vanishing point) 311, the lower sight line position 312, and the uppersight line position 313 illustrated in FIG. 10 are described above inthe first embodiment, a description of them is omitted herein.

FIG. 11B illustrates another example 1 in the second embodiment. In theexample in FIG. 11B, a virtual line 703 b projecting from a singlepresentation area 701 b corresponding to the first object is connectedto a virtual line 704 b projecting from a single presentation area 702 bcorresponding to the second object via the basic sight line position 311illustrated in FIG. 10.

FIG. 11C illustrates another example 2 in the second embodiment. In theexample in FIG. 11C, a virtual line 703 c projecting from a singlepresentation area 701 c corresponding to the first object is connectedto a virtual line 704 c projecting from a single presentation area 702 ccorresponding to the second object via the lower sight line position 312illustrated in FIG. 10.

FIG. 11D illustrates another example 3 in the second embodiment. In theexample in FIG. 11B, a virtual line 703 d projecting from a singlepresentation area 701 d corresponding to the first object is connectedto a virtual line 704 d projecting from a single presentation area 702 dcorresponding to the second object via the upper sight line position 313illustrated in FIG. 10. Since the upper sight line position 313 ispositioned so as not to be overlapped with the preceding vehicle in FIG.11D, the virtual lines 703 d and 704 d are not overlapped and displayedon the preceding vehicle.

The single presentation areas and the virtual lines illustrated in FIG.11B to FIG. 11D are determined by the determination unit 101 in StepsS503 and S504 in FIG. 9, as in the single presentation areas and thevirtual lines illustrated in FIG. 11A. Specifically, the determinationunit 101 determines the single presentation areas described above on thebasis of the position information about the first object, the positioninformation about the second object, which are acquired, the basic sightline position 311, the lower sight line position 312, and the uppersight line position 313 and determines the virtual lines described inany of the examples.

The shapes of the virtual lines are not limited to the ones illustratedin FIG. 11A to FIG. 11D in the second embodiment. For example, when thepreceding vehicle exists on the front side of the view of the driver,the virtual lines may have a shape allowing the positions of the objectsto be connected so as not to be overlapped with the preceding vehicle oronly part of the virtual lines which is overlapped with the precedingvehicle may not be displayed. This avoids the difficulty in the visualrecognition of the preceding vehicle by the user because of the virtuallines that are overlapped and displayed on the preceding vehicle.

In the second embodiment, the virtual lines illustrated in FIG. 11A toFIG. 11D may be selectively used depending on the status of the frontside of the vehicle. For example, the virtual lines 703 a and 704 aillustrated in FIG. 11A may be displayed when no preceding vehicleexists, the virtual lines 703 c and 704 c illustrated in FIG. 11C may bedisplayed when the vehicular gap between the own vehicle and thepreceding vehicle is sufficiently kept, and the virtual lines 703 d and704 d illustrated in FIG. 11D may be displayed when the vehicular gapbetween the own vehicle and the preceding vehicle is not sufficientlykept.

In the second embodiment, the virtual lines illustrated in FIG. 11A toFIG. 11D, that is, the presentation area corresponding to the firstobject and the presentation area corresponding to the second object (forexample, the presentation areas 703 a and 704 a, the presentation areas703 b and 704 b, the presentation areas 703 c and 704 c, and thepresentation areas 703 d and 704 d) are not necessarily connected toeach other. For example, a tip of the presentation area corresponding tothe first object toward the certain position may be close to a tip ofthe presentation area corresponding to the second object toward thecertain position at the certain position in the foreground 300. Thedistance between the tips is determined in advance. The certain positionis not limited to the basic sight line position 311, the lower sightline position 312, and the upper sight line position 313. However, thecertain position is desirably close to the normal sight line position(for example, the basic sight line position 311) during the driving fromsafety aspect.

Although the size of the single presentation area is determined on thebasis of the size of the object (the size information about the object)in the second embodiment, the size of the single presentation area maybe determined on the basis of the distance between the object and theown vehicle (distance information) or at least one of the speed of theobject, the speed of the own vehicle, and the relative speed (speedinformation) between the object and the own vehicle.

In the second embodiment, when it is not possible to present all thevirtual lines under restrictions of, for example, the size of thedisplay, part of the virtual lines may be presented in a range in whichthe virtual lines are capable of being presented.

Although the single presentation areas and the virtual lines areoverlapped and displayed on the foreground which the driver is capableof visually recognizing in the second embodiment, the singlepresentation areas and the virtual lines are not limited to this. Forexample, the single presentation areas and the virtual lines may beoverlapped and displayed on the image data of the foreground which thedriver is capable of visually recognizing.

Although the representative information added to the single presentationareas and the virtual lines is a color in the second embodiment, thepresentation areas may be translucent so that the driver is capable ofvisually recognizing the background of the presentation areas in thiscase. The representative information may be a shape. The shape of thesingle presentation area is set to, for example, a rectangle or a starshape depending on the type of the object.

As described above, the display control apparatus 100 of the secondembodiment has the characteristics of the first embodiment and ischaracterized in that the size of the single presentation area isdetermined on the basis of the size of the object. Accordingly, thedisplay control apparatus 100 achieves the advantages of the firstembodiment and is capable of causing the driver to feel the sense ofdistance between the object and the own vehicle, specifically, todetermine whether the object is approaching the own vehicle or is movingapart from the own vehicle.

Third Embodiment

A third embodiment of the present disclosure will now be described.

A phenomenon called attentional blink is known as a visual feature ofhuman beings. The attentional blink is a phenomenon in which attentionis made temporarily unavailable, like a case in which human beings aretemporarily blinded to the external world when they blink. The driver isnot capable of paying attention to the second object during anattentional dwell time after the driver pays attention to the firstobject in the foreground. Although the attentional dwell time is, forexample, 0.6 seconds to 0.7 seconds, the attentional dwell time may haveanother value due to individual differences. If the informationcorresponding to the second object is presented, for example, within 0.6seconds after the information corresponding to the first object ispresented when the attentional dwell time is 0.6 seconds, the driver maypossibly overlook either of the objects. In the ADAS, when theinformation about an object to which attention should be paid (forexample, the position information and the size information) is acquired,it is necessary to present the information (for example, thepresentation area) corresponding to the object as soon as possible.Accordingly, it may be considered that the presentation areas formultiple objects are presented substantially simultaneously with theacquisition of the position information and the size information aboutthe respective objects (the timing when the position information and thesize information about the respective objects are acquired ishereinafter referred to as information acquisition timing). In thesecond embodiment, the determination of the virtual lines when theposition information and the size information about multiple objects areacquired in a time within the attentional dwell time allows the driverto recognize the multiple objects without causing the overlook duringthe attentional blink.

The block diagram illustrating an exemplary configuration of a videosystem according to the third embodiment is the same as that of thefirst embodiment illustrated in FIG. 1. Referring to FIG. 1, the videosystem 10 includes the display control apparatus 100, the display source400, and the recognition unit 200. The display control apparatus 100includes the determination unit 101 and the control unit 102. Theoperations of the recognition unit 200 and the display source 400 aresame as those in the first embodiment.

An exemplary operation of the display control apparatus 100 of the thirdembodiment will now be described. FIG. 12 is a flowchart illustrating anexemplary operational process of the display control apparatus 100 ofthe third embodiment. The “object” in the third embodiment has the samemeaning as that of the “moving body” in the first embodiment. The userof the display control apparatus 100 is the driver of a vehicle in thethird embodiment.

A timing chart illustrated in FIG. 13 and an example of the foregroundillustrated in FIG. 10 are used in the description of the exemplaryoperation.

FIG. 13 is the timing chart indicating an example of the informationacquisition timing of an object by the determination unit 101 during atime period from a time t1 to a time t4. The horizontal axis representstime and the vertical axis indicates whether the information is acquiredin FIG. 13. When a rectangular wave has a high value on the verticalaxis, the position information and the size information are acquired andthe object corresponding to the position information and the sizeinformation exists in the foreground. In contrast, when the rectangularwave has a low value on the vertical axis, the position information andthe size information are not acquired and the object corresponding tothe position information and the size information does not exist in theforeground. Referring to FIG. 13, reference numeral d701 denotes theinformation acquisition timing of the position information and the sizeinformation about the first object and reference numeral d702 denotesthe information acquisition timing of the position information and thesize information about the second object. Similarly, reference numerald703 denotes the information acquisition timing of the positioninformation and the size information about a third object and referencenumeral d704 denotes the information acquisition timing of the positioninformation and the size information about a fourth object.

FIG. 14A, FIG. 14B, and FIG. 14C illustrate examples of the foregroundthat is varied with the progress of the vehicle in time series. FIG. 14Aillustrates an example of how the presentation area is presented in aforeground 1000 a at the time t1. A presentation area 1001 a is thesingle presentation area corresponding to the first object. The firstobject is, for example, a pedestrian who is walking on the left side inthe traveling direction of the vehicle.

FIG. 14B illustrates an example of how the presentation areas arepresented in a foreground 1000 b at the time t3. In this example, thefollowing presentation areas are presented, in addition to thepresentation area 1001 a described above. A presentation area 1002 a isthe single presentation area corresponding to the second object and apresentation area 1002 b is the virtual line projecting from thepresentation area 1002 a. The second object is, for example, apedestrian who is walking on the left side in the traveling direction ofthe vehicle. A presentation area 1003 a is the single presentation areacorresponding to the third object and a presentation area 1003 b is thevirtual line projecting from the presentation area 1003 a. The thirdobject is, for example, a pedestrian who is walking on the right side inthe traveling direction of the vehicle. A line segment 1010 is astraight line connecting the basic sight line position (refer toreference numeral 311 in FIG. 3 and FIG. 10) to the own vehicle. A point1011 is on the line segment 1010 and is at a position where the sum ofthe distance from the position of the first object, the distance fromthe position of the second object, and the distance from the position ofthe third object is minimized. The line segment 1010 and the point 1011are virtual objects on which the display of the virtual lines is basedand are not displayed on the display.

FIG. 14C illustrates an example of how the presentation areas arepresented in a foreground 1000 c at the time t4. In this example, thepresentation of the presentation area 1001 a described above disappearsand a presentation area 1004 a is newly presented. In other words, thisexample indicates a case in which the first object disappears from theforeground and the fourth object newly appears in the foreground. Thepresentation area 1004 a is the single presentation area correspondingto the fourth object. The fourth object is, for example, a pedestrianwho is walking on the left side in the traveling direction of thevehicle. In the example in FIG. 14C, the point 1011 is on the linesegment 1010 and is at a position where the sum of the distance from theposition of the second object, the distance from the position of thethird object, and the distance from the position of the fourth object isminimized.

The flowchart illustrating an exemplary operational process of thedisplay control apparatus 100 of the second embodiment will now bedescribed.

Referring to FIG. 12, in Step S801, the determination unit 101 acquiresthe position information and the size information about the object fromthe recognition unit 200. For example, at the time t1, the determinationunit 101 acquires the position information and the size informationabout the first object. For example, at the time t2, the determinationunit 101 acquires the position information and the size informationabout the second object. For example, at the time t3, the determinationunit 101 acquires the position information and the size informationabout the third object. For example, at the time t4, the determinationunit 101 acquires the position information and the size informationabout the fourth object. The position information and the sizeinformation are, for example, the frames surrounding the objects (forexample, refer to the frames 601 and 602 in FIG. 10).

In Step S802, the determination unit 101 stores a time (an informationacquisition time) when the position information and the size informationabout the object is acquired. For example, when the position informationand the size information about the first object are acquired, thedetermination unit 101 stores the time t1 when the position informationand the size information about the first object are acquired. Forexample, when the position information and the size information aboutthe second object are acquired, the determination unit 101 stores thetime t2 when the position information and the size information about thesecond object are acquired. For example, when the position informationand the size information about the third object are acquired, thedetermination unit 101 stores the time t3 when the position informationand the size information about the third object are acquired. Forexample, when the position information and the size information aboutthe fourth object are acquired, the determination unit 101 stores thetime t4 when the position information and the size information about thefourth object are acquired.

In Step S803, the determination unit 101 determines whether thepresentation area that is being presented exists. If the determinationunit 101 determines that no presentation area that is being presentedexists (NO in Step S803), the process goes to Step S805. For example,when the position information and the size information about the firstobject are acquired, the process goes to Step S805 because nopresentation area that is being presented exists. If the determinationunit 101 determines that the presentation area that is being presentedexists (YES in Step S803), the process goes to Step S804. For example,when the position information and the size information about each of thesecond object, the third object, and the fourth object are acquired, theprocess goes to Step S804 because the presentation area that is beingpresented exists.

In Step S804, the determination unit 101 determines whether thedifference between the information acquisition times is shorter than acertain time. The certain time is the attentional dwell time describedabove and is, for example, 0.6 seconds. However, the certain time is notlimited to this. For example, at the time t2, the determination unit 101determines whether the difference between the time t2 and the time t1(Δt12 in FIG. 13) is shorter than the certain time. For example, at thetime t3, the determination unit 101 determines whether the differencebetween the time t3 and the time t2 (Δt23 in FIG. 13) is shorter thanthe certain time. For example, at the time t4, the determination unit101 determines whether the difference between the time t4 and the timet3 (Δt34 in FIG. 13) is shorter than the certain time.

If the determination unit 101 determines that the difference between theinformation acquisition times is longer than or equal to the certaintime (NO in Step S804), the process goes to Step S805. For example, atthe time t2, the determination unit 101 determines that the differencebetween the time t2 and the time t1 is longer than or equal to thecertain time. For example, at the time t4, the determination unit 101determines that the difference between the time t4 and the time t3 islonger than or equal to the certain time.

If the determination unit 101 determines that the difference between theinformation acquisition times is shorter than the certain time (YES inStep S804), the process goes to Step S806. For example, at the time t3,the determination unit 101 determines that the difference between thetime t3 and the time t2 is shorter than the certain time.

In Step S805, the determination unit 101 determines the singlepresentation area on the basis of the position information and the sizeinformation that are acquired. For example, if the determination unit101 determines that no presentation area that is being presented exists(NO in Step S803), the determination unit 101 determines the singlepresentation area corresponding to the first object on the basis of theposition information and the size information about the first object.This single presentation area is, for example, the presentation area1001 a in FIG. 14A. For example, if the determination unit 101determines that the difference between the time t2 and the time t1 islonger than or equal to the certain time (NO in Step S804), thedetermination unit 101 determines the single presentation areacorresponding to the second object on the basis of the positioninformation and the size information about the second object. Thissingle presentation area is, for example, the presentation area 1002 ain FIG. 14B. For example, if the determination unit 101 determines thatthe difference between the time t4 and the time t3 is longer than orequal to the certain time (NO in Step S804), the determination unit 101determines the single presentation area corresponding to the fourthobject on the basis of the position information and the size informationabout the fourth object. This single presentation area is, for example,the presentation area 1004 a in FIG. 14C.

In Step S806, the determination unit 101 determines the connectedpresentation areas for connecting the position of the third object tothe position of the second object on the basis of the positioninformation and the size information that are acquired. For example, ifthe determination unit 101 determines that the difference between thetime t3 and the time t2 is shorter than the certain time (YES in StepS804), the determination unit 101 determines the single presentationarea corresponding to the third object and also determines the virtuallines connecting the position of the third object to the position of thesecond object. A specific example of this operation will now bedescribed.

First, the determination unit 101 determines the single presentationarea corresponding to the third object on the basis of the positioninformation and the size information about the third object. This singlepresentation area is, for example, the presentation area 1003 a in FIG.14B.

Then, the determination unit 101 identifies the point 1011 where the sumof the distances from the positions of all the objects existing in theforeground is minimized on the line segment 1010. For example, in thecase of the foreground 1000 b illustrated in FIG. 14B, the determinationunit 101 identifies the point 1011 where the sum of the distances fromthe positions of the first object, the second object, and the thirdobject is minimized on the line segment 1010. For example, in the caseof the foreground 1000 c illustrated in FIG. 14C, the determination unit101 identifies the point 1011 where the sum of the distances from thepositions of the second object, the third object, and the fourth objectis minimized on the line segment 1010.

Then, the determination unit 101 determines the virtual line so as toproject from the single presentation area corresponding to the secondobject to the point 1011. This virtual line is, for example, thepresentation area 1002 b illustrated in FIG. 14B and FIG. 14C.

Similarly, the determination unit 101 determines the virtual line so asto project from the single presentation area corresponding to the thirdobject to the point 1011. This virtual line is, for example, thepresentation area 1003 b illustrated in FIG. 14B and FIG. 14C.

The presentation area 1002 a is connected to the presentation area 1003a via the point 1011 in the above manner. In other words, thepresentation area 1002 a and the presentation area 1003 a are determinedto be the virtual lines connecting the position of the third object tothe position of the second object.

In Step S807, the control unit 102 adds the representative informationto the presentation area determined by the determination unit 101. Then,the representative information is supplied to the display source 400where the representative information is projected on a certain displayas the virtual image. A predetermined color, red, is added to the singlepresentation areas and the virtual lines as the representativeinformation about the pedestrian as an example here. For example, inFIG. 14A to FIG. 14C, the presentation areas 1001 a, 1002 a, 1003 a, and1004 a, and the virtual lines 1002 b and 1003 b are displayed in red.

The shapes of the virtual lines are not limited to the ones illustratedin FIG. 14A to FIG. 14C in the third embodiment. For example, when thepreceding vehicle exists on the front side of the foreground, thevirtual lines may have a shape allowing the positions of the objects tobe connected to each other so as not to be overlapped with the precedingvehicle, only part of the virtual lines which is overlapped with thepreceding vehicle may not be displayed, or a point on the line segment(for example, the point 1011) may be positioned so as not to beoverlapped with the preceding vehicle and the virtual lines may have ashape allowing the position of the object to be connected to the pointon the line segment. This avoids the difficulty in the visualrecognition of the preceding vehicle by the user because of the virtuallines that are overlapped and displayed on the preceding vehicle.

Although the determination unit 101 determines whether the differencebetween the information acquisition times of the two objects is shorterthan the certain time in the third embodiment, the time compared withthe certain time is not limited to the difference between theinformation acquisition times of the two objects. For example, the timecompared with the certain time may be the difference between theinformation acquisition time of the first object and the time when theown vehicle is estimated to come close to the second object within acertain distance from the traveling speed and the position when thesecond object is the object most closest to the own vehicle.

Although the determination unit 101 determines whether the differencebetween the information acquisition times of the two objects appearingin the foreground is shorter than the certain time in the thirdembodiment, the time compared with the certain time is not limited tothe difference between the information acquisition times of the twoobjects appearing in the foreground. For example, the objects may belimited to the object the moving direction of which is varied, such as apedestrian who is moving along the sidewalk and starts to cross theroad, or the object the moving speed of which is varied, such as abicycle that is moving slowly at a walking speed and starts to move at ahigher speed. Priority may be given to these objects over other objects.

Although the size of the single presentation area is varied depending onthe size of the object in the third embodiment, the size of the singlepresentation area may be varied depending on the distance between theobject and the own vehicle or at least one of the speed of the object,the speed of the own vehicle, and the relative speed between the objectand the own vehicle.

In the third embodiment, when it is not possible to present all thevirtual lines under restrictions of, for example, the size of thedisplay, part of the virtual lines may be presented in a range in whichthe virtual lines are capable of being presented.

Although the single presentation areas and the virtual lines areoverlapped and displayed on the foreground which the driver is capableof visually recognizing in the third embodiment, the single presentationareas and the virtual lines are not limited to this. For example, thesingle presentation areas and the virtual lines may be overlapped anddisplayed on the image data of the foreground which the driver iscapable of visually recognizing.

Although the representative information added to the single presentationareas and the virtual lines is a color in the third embodiment, thepresentation areas may be translucent so that the driver is capable ofvisually recognizing the background of the presentation areas in thiscase. The representative information may be a shape. The shape of thesingle presentation area is set to, for example, a rectangle or a starshape depending on the type of the object.

As described above, the display control apparatus 100 of the thirdembodiment has the characteristics of the second embodiment and ischaracterized in that, when the position information and the sizeinformation about multiple objects are acquired in a time shorter thanthe attentional dwell time, the virtual lines connecting the objects aredetermined. Accordingly, the display control apparatus 100 achieves theadvantages of the second embodiment and is capable of causing the driverto recognizing the multiple objects without causing the overlook duringthe attentional blink.

Fourth Embodiment

A fourth embodiment of the present disclosure will now be described.Although the “object” in the second and third embodiments means only themoving body, the “object” in the fourth embodiment includes an objectfixed to the road environment, in addition to the moving body. The userof the display control apparatus 100 is the driver of a vehicle in thefourth embodiment.

Phenomena called feature search and conjunction search are known asvisual features of human beings. The feature search is search in whichan object having a single feature, such as a color or an orientation, isvisually searched for and in which the feature is capable of beingsubjected to spatial parallel processing. In contrast, the conjunctionsearch is search in which an object having multiple features, such as acolor and an orientation, is visually searched for. The object isdetermined by paying visual attention to the object and, therefore,serial processing is performed in the conjunction search. As apparentfrom such phenomena, when the number of objects that are moving(hereinafter referred to as moving objects), such as pedestrians and thepreceding vehicle, is increased and/or the number of objects fixed tothe road environment (hereinafter referred to as fixed objects), such asa traffic signal and a road sign, is increased to reduce the ease ofrecognition of the foreground, it is necessary to ensure a sufficienttime to successively pay attention to the multiple objects with theattentional dwell time sandwiched therebetween. The display controlapparatus 100 of the fourth embodiment determines the presentation areaof the object on the basis of the ease of recognition of the foregroundand, if the ease of recognition of the foreground is reduced, promptsthe driver to decelerate the own vehicle by audio or the like to ensurethe time when the driver pays attention, thus causing the driver to morereliably recognize the multiple objects.

The block diagram illustrating an exemplary configuration of a videosystem according to the fourth embodiment is the same as that of thefirst embodiment illustrated in FIG. 1. Referring to FIG. 1, the videosystem 10 includes the display control apparatus 100, the display source400, and the recognition unit 200. The display control apparatus 100includes the determination unit 101 and the control unit 102, as in thefirst embodiment. The operations of the recognition unit 200 and thedisplay source 400 are same as those in the first embodiment.

An exemplary operation of the display control apparatus 100 of thefourth embodiment will now be described. FIG. 15 is a flowchartillustrating an exemplary operational process of the display controlapparatus 100 of the fourth embodiment.

An example of the foreground illustrated in FIG. 16 is used in thedescription of the exemplary operation.

For example, a road, buildings, and pedestrians walking on both sides ofthe road exist in a foreground 1200 in FIG. 16 as a scene on the frontside of the vehicle. Referring to FIG. 16, a frame 1201 indicates theposition and the size of a pedestrian (an example of the first object)who is walking on the left side in the traveling direction of thevehicle and a frame 1202 indicates the position and the size of apedestrian (an example of the second object) who is walking on the rightside in the traveling direction of the vehicle (refer to rectangles inFIG. 16). The pedestrians are examples of the moving objects.

Referring to FIG. 16, a frame 1211 indicates the position and the sizeof a cross-walk, a frame 1212 indicates the position and the size of atraffic signal, and a frame 1213 indicates the position and size of atraffic sign (refer to ellipsis and a trapezoid drawn with broken linesin FIG. 16). The cross-walk, the traffic signal, and the traffic signare examples of the fixed objects.

Although a single presentation area 1201 a corresponding the firstobject, a single presentation area 1202 a corresponding to the secondobject, and virtual lines 1201 b and 1202 b connecting the position ofthe first object to the position of the second object are illustrated inFIG. 16 for convenience, in addition to the above frames, the singlepresentation area 1201 a, the single presentation area 1202 a, and thevirtual lines 1201 b and 1202 b are displayed after Step S1107 in FIG.15 described below.

The flowchart illustrating the exemplary operational process of thedisplay control apparatus 100 of the fourth embodiment in FIG. 15 willnow be described in detail.

Referring to FIG. 15, in Step S1101, the determination unit 101 acquiresthe position information and the size information about each of thefixed objects and the moving objects from the recognition unit 200. Forexample, in the case of the foreground 1200, the frames 1201 and 1202are acquired as the position information and the size information aboutthe moving objects and the frames 1211, 1212, and 1213 are acquired asthe position information and the size information about the fixedobjects.

In Step S1102, the determination unit 101 calculates the sum of thenumbers of the objects the position information and the size informationof which are acquired and the sum of the sizes (areas) of the objectsthe position information and the size information of which are acquired.For example, in the case of the foreground 1200, since the frames thatare acquired are the five frames 1201, 1202, 1211, 1212, and 1213, thesum of the numbers of the objects is five. For example, in the case ofthe foreground 1200, the sum of the areas of the five frames 1201, 1202,1211, 1212, and 1213 that are acquired is equal to the sum of the sizesof the objects.

In Step S1103, the determination unit 101 determines whether the sum ofthe numbers of the objects is smaller than a certain number. If thedetermination unit 101 determines that the sum of the numbers of theobjects is greater than or equal to the certain number (NO in StepS1103), the process goes to Step S1105. The driver is in a state inwhich it is difficult for the driver to recognize the objects in theforeground when the sum of the numbers of the objects is greater than orequal to the certain number. If the determination unit 101 determinesthat the sum of the numbers of the objects is smaller than the certainnumber (YES in Step S1103), the process goes to Step S1104. The driveris in a state in which it is easy for the driver to recognize theobjects in the foreground when the sum of the numbers of the objects issmaller the certain number.

In Step S1104, the determination unit 101 determines whether the sum ofthe sizes of the objects is smaller than a certain size. If thedetermination unit 101 determines that the sum of the sizes of theobjects is greater than or equal to the certain size (NO in Step S1104),the process goes to Step S1105. The driver is in the state in which itis difficult for the driver to recognize the objects in the foregroundwhen the sum of the sizes of the objects is greater than or equal to thecertain size. If the determination unit 101 determines that the sum ofthe sizes of the objects is smaller than the certain size (YES in StepS1104), the process goes to Step S1106. The driver is in the state inwhich it is easy for the driver to recognize the objects in theforeground when the sum of the sizes of the objects is smaller thecertain size.

In Step S1105, since the driver is in the state in which it is difficultfor the driver to recognize the objects in the foreground, thedetermination unit 101 does not determine the presentation area anddetermines audio deceleration guidance. In this case, for example, audioto prompt the driver to decelerate the vehicle is output from a speaker(not illustrated).

In Step S1106, the determination unit 101 determines the connectedpresentation areas including the virtual lines and the singlepresentation area, such as the one described above in Step S805 in FIG.12. Since the detailed description of Step S1106 has been given in StepS504 in FIG. 9 and Step S806 in FIG. 12, the description is omittedherein. In this case, for example, the single presentation area 1201 acorresponding to the first object, the single presentation area 1202 acorresponding to the second object, and the virtual lines 1201 b and1202 b connecting the position of the first object to the position ofthe second object are displayed in the foreground 1200 in FIG. 16.Although the example of the presentation in FIG. 16 is the same as thosein FIG. 14B and FIG. 14C in the third embodiment, the presentation isnot limited to the above one. Any of the examples of the presentation inFIG. 11A to FIG. 11D in the second embodiment may be used.

Since Step S1107 is the same as Step S505 in FIG. 9 and Step S807 inFIG. 12, the description of Step S1107 is omitted herein.

Although the ease of recognition of the foreground is determined on thebasis of the number and the sizes of the objects in the fourthembodiment, the determination of the ease of recognition of theforeground is not limited to this. For example, the ease of recognitionof the foreground may be determined on the basis of, for example, thecolors, the shapes, or the moving directions of the objects.

Although the sum of the number of the fixed objects and the number ofthe moving objects is used in the fourth embodiment, either of thenumber of the fixed objects and the number of the moving objects may beused or the objects may be weighted depending on the sizes of theobjects. Although the cross-walk, the traffic signal, and the trafficsign are used as the fixed objects, a combination of any one kind ormultiple kinds of the cross-walk, the traffic signal, and the trafficsign may be used. The fixed objects may include, for example, a curvedmirror, a utility pole, and a signboard, in addition to the cross-walk,the traffic signal, and the traffic sign.

Although the total number of the sum of the sizes of the fixed objectsand the sum of the sizes of the moving objects is used in the fourthembodiment, the total number of the sum of the sizes of the fixedobjects and the sum of the sizes of the moving objects is not limitedlyused. For example, the sum of the sizes of either of the fixed objectsand the moving objects may be used or the objects may be weighteddepending on the kinds or the approach speeds of the objects.

Although the presentation area is not determined and the audiodeceleration guidance is performed in the fourth embodiment, the audiodeceleration guidance may be performed, in addition to the display ofthe presentation area. Alternatively, a color or the like indicating thestatus in which the foreground is difficult to be recognized may beadded to the presentation area. Assistance other than the audioassistance or assistance other than the deceleration guidance may beperformed.

As described above, the display control apparatus 100 of the fourthembodiment has the characteristics of the second and third embodimentsand is characterized in that the virtual lines (including the singlepresentation area) are presented when it is easy for the driver torecognize the foreground and the audio deceleration guidance isperformed when it is difficult for the driver to recognize theforeground. Accordingly, the display control apparatus 100 achieves theadvantages of the second and third embodiments and is capable ofensuring the time in which the driver pays attention and causing thedriver to reliably recognize the multiple objects to which attentionshould be paid without requiring the sight line shift.

Although the first to fourth embodiments of the present disclosure aredescribed above with reference to the drawings, the function of eachcomponent in the apparatuses described above may be realized by acomputer program.

FIG. 17 is a block diagram illustrating an exemplary hardwareconfiguration of a computer, which realizes the function of eachcommunication using the program. Referring to FIG. 17, a computer 1300includes an input unit 1301, such as a keyboard, a mouse, and/or a touchpad; an output unit 1302 such as a display and/or a speaker; a centralprocessing unit (CPU) 1303; a read only memory (ROM) 1304; a randomaccess memory (RAM) 1305; a memory unit 1306 such as a hard disk deviceor a solid state drive (SSD); a reading unit 1307 that reads informationfrom a recording medium, such as a digital versatile disk-read onlymemory (DVD-ROM) or a universal serial bus (USB) memory; and atransmission and reception unit 1308 that performs communication via anetwork. The above components are connected to each other via a bus1309.

The reading unit 1307 reads out the program from the recording medium inwhich the program for realizing the function of each component isrecorded and stores the program in the memory unit 1306. Alternatively,the transmission and reception unit 1308 communicates with a serverapparatus connected to the network to store the program for realizingthe function of each component, which is downloaded from the serverapparatus, in the memory unit 1306.

The CPU 1303 copies the program stored in the memory unit 1306 into theRAM 1305 and sequentially reads out instructions included in the programfrom the RAM 1305 for execution to realize the function of eachcomponent. In the execution of the program, the pieces of informationresulting from the various processes described in the above embodimentsare stored in the RAM 1305 and the memory unit 1306 and areappropriately used.

The program to be executed in the above components may be recorded in aportable computer-readable recording medium (for example, a DVD, a USBmemory, a memory card, or the like) in an installable-format file or anexecutable-format file for provision. Alternatively, the program may bestored in a computer connected to a network, such as the Internet, andmay be downloaded to a terminal via the network. Alternatively, theprogram may be incorporated into a storage unit (for example, a ROM) inthe terminal in advance for provision.

The present disclosure is not limited to the first to fourth embodimentsof the present disclosure described above, and various changes andmodifications may be made to the present disclosure without departingfrom the spirit and scope thereof.

For example, although the example is described in the above embodimentsin which the user who visually recognizes the presentation areas is thedriver of the vehicle, the user may be a pedestrian. For example, theuser may be walking (or running) while wearing the video system 10 onwhich the HMD is mounted to achieve the advantages described in theabove embodiments.

The present disclosure is useful for technologies concerning, forexample an information presentation apparatus, an informationpresentation method, an information presentation program, and aprojection apparatus, which present information for alerting the driverof a vehicle.

What is claimed is:
 1. A method for a video system including recognitioncircuitry that recognizes one or more moving bodies existing on a frontside of a vehicle that a user drives, and a display source that displaysa certain image generated on the basis of a result of the recognition bythe recognition circuitry on a display, the method comprising:determining whether the one or more moving bodies includes two or moremoving bodies that exist on the front side of the vehicle on the basisof input data from the recognition circuitry; controlling the displaysource, when it is determined that two or more moving bodies exist onthe front side of the vehicle, so as to generate the certain imagerepresenting one or more virtual lines connecting the corresponding twoor more moving bodies from a certain point; and displaying the generatedcertain image on the display, wherein a position of the certain point ischanged on the basis of a status of the front side of the vehicleincluded in the result of the recognition, wherein the status of thefront side of the vehicle is that a preceding vehicle exists or not onthe front side of the vehicle, and wherein, when the preceding vehicleis on the front side of the vehicle, the position of the certain pointis changed not to overlap with the preceding vehicle.
 2. The methodaccording to claim 1, further comprising: when it is determined that anew moving body exists on the front side of the vehicle on the basis ofan input data from the recognition circuitry, the display source iscontrolled so as to generate the certain image representing a virtualline connecting the new moving body from the certain point and displaythe generated certain image on the display.
 3. The method according toclaim 1, further comprising: when it is determined that at least onemoving body, among the two or more moving bodies, disappears from thefront side of the vehicle on the basis of an input data from therecognition circuitry, the display source is controlled so as togenerate the certain image in which at least one virtual line includedin the one or more virtual lines connected with the corresponding atleast one moving body that is determined to disappear from the frontside of the vehicle is deleted and display the generated certain imageon the display.
 4. The method according to claim 1, wherein a thicknessof each of the one or more virtual lines is determined on the basis ofat least one of a distance from the vehicle to the corresponding movingbody, a speed of the vehicle, and a relative speed between thecorresponding vehicle and any one of the one or more moving bodies. 5.The method according to claim 1, wherein the video system includes acamera that detects at least one of an orientation and a line of sightof the user, and wherein the method further includes determining theposition of the certain point on the basis of the orientation and theline of sight detected by the camera.
 6. The method according to claim1, wherein, when it is determined that two or more moving bodies exist,the display source is controlled so as to generate the certain imagethat represents the one or more virtual lines connecting each of thecorresponding two or more moving bodies from the certain point and thecertain image that represents a virtual graphic of a certain shape at aposition corresponding to each of the two or more moving bodies anddisplay the generated certain images on the display.
 7. A displaycontrol apparatus in a video system including recognition circuitry thatrecognizes one or more moving bodies existing on a front side of avehicle that a user drives, and a display source that displays a certainimage generated on the basis of a result of the recognition by therecognition circuitry on a display, the display control apparatuscomprising: determination circuitry that determines whether the one ormore moving bodies includes two or more moving bodies that exist on thefront side of the vehicle on the basis of input data from therecognition circuitry; and a controller that, when the determinationcircuitry determines that the two or more moving bodies exist on thefront side of the vehicle, controls the display source so as to generatethe certain image representing one or more virtual lines connecting eachof the corresponding two or more moving bodies from a certain point, andso as to display the generated certain image on the display, wherein thecontroller changes a position of the certain point on the basis of astatus of the front side of the vehicle included in the result of therecognition, wherein the status of the front side of the vehicle is thata preceding vehicle exists or not on the front side of the vehicle, andwherein, when the preceding vehicle is on the front side of the vehicle,the controller changes the position of the certain point not to overlapwith the preceding vehicle.
 8. The display control apparatus accordingto claim 7, wherein, when the determination circuitry determines that anew moving body exists on the front side of the vehicle on the basis ofan input data from the recognition circuitry, the controller controlsthe display source so as to generate the certain image representing anew virtual line connecting the new moving body from the certain pointand display the generated certain image on the display.
 9. The displaycontrol apparatus according to claim 7, wherein, when the determinationcircuitry determines that at least one moving body, among the two ormore moving bodies, disappears from the front side of the vehicle on thebasis of input data from the recognition circuitry, the controllercontrols the display source so as to generate the certain image in whichat least one virtual line included in the one or more virtual linesconnected with the corresponding at least one moving body that isdetermined to disappear from the front side of the vehicle is deletedand display the generated certain image on the display.
 10. The displaycontrol apparatus according to claim 7, wherein a thickness of each ofthe one or more virtual lines is determined on the basis of at least oneof a distance from the vehicle to the corresponding moving body, a speedof the vehicle, and a relative speed between the corresponding vehicleand any one of the moving bodies.
 11. The display control apparatusaccording to claim 7, wherein the video system includes a camera thatdetects at least one of an orientation and a line of sight of the user,and wherein the controller determines the position of the certain pointon the basis of the orientation and the line of sight detected by thecamera.
 12. The display control apparatus according to claim 7, wherein,when the determination circuitry determines that two or more movingbodies exist, the controller controls the display source so as togenerate the certain image that represents the one or more virtual linesconnecting the corresponding two or more moving bodies from the certainpoint and the certain image that represents a virtual graphic of acertain shape at a position corresponding to each of the two or moremoving bodies and display the generated certain images on the display.13. A vehicle, comprising: the display control apparatus according toclaim
 7. 14. A non-transitory, tangible computer-readable recordingmedium recording a display control program causing a computer in a videosystem including recognition circuitry that recognizes one or moremoving bodies existing on a front side of a vehicle that a user drives,and a display source that displays a certain image generated on thebasis of a result of the recognition by the recognition circuitry on adisplay to execute: determining whether the one or more moving bodiesincludes two or more moving bodies that exist on the front side of thevehicle on the basis of input data from the recognition circuitry;controlling the display source, when it is determined that two or moremoving bodies exist on the front side of the vehicle, so as to generatethe certain image representing one or more virtual lines connecting eachof the corresponding two or more moving bodies from a certain point, andso as to display the generated certain image on the display, whereinchanging a position of the certain point is performed on the basis of astatus of the front side of the vehicle included the result of therecognition, wherein the status of the front side of the vehicle is thata preceding vehicle exists or not on the front side of the vehicle, andwherein, when the preceding vehicle is on the front side of the vehicle,changing the position of the certain point not to overlap with thepreceding vehicle.
 15. A display control apparatus, comprising: firstconnecting circuitry connected to recognition circuitry that recognizesone or more moving bodies existing on the front side of a vehicle that auser drives: second connecting circuitry connected to a display sourcethat displays a certain image generated on the basis of a result of therecognition by the recognition circuitry on a display; determinationcircuitry that determines whether the one or more moving bodies includestwo or more moving bodies that exist on the front side of the vehicle onthe basis of input data from the recognition circuitry; and a controllerthat, when the determination circuitry determines that the two or moremoving bodies exist on the front side of the vehicle, controls thedisplay source so as to generate the certain image representing one ormore virtual lines connecting each of the corresponding two or moremoving bodies from a certain point, and displays the generated certainimage including the one or more virtual lines connecting each of thecorresponding two or more moving bodies on the display, wherein thecontroller changes a position of the certain point on the basis of astatus of the front side of the vehicle included the result of therecognition, wherein the status of the front side of the vehicle is thata preceding vehicle exists or not on the front side of the vehicle, andwherein, when the preceding vehicle is on the front side of the vehicle,the controller changes the position of the certain point not to overlapwith the preceding vehicle.